With element.disabled = true;
XPages Check Box Group generates a <table> and you might be able to disable the table.
If that does not work, all checkboxes in the group have the same name and you can use dojo.query to get them all. Then just iterate the check box collection and disable them.